The Cell Tracking Challenge: 10 years of objective benchmarking.

The Cell Tracking Challenge is an ongoing benchmarking initiative that has become a reference in cell segmentation and tracking algorithm development. Here, we present a significant number of improvements introduced in the challenge since our 2017 report. These include the creation of a new segmentation-only benchmark, the enrichment of the dataset repository with new datasets that increase its diversity and complexity, and the creation of a silver standard reference corpus based on the most competitive results, which will be of particular interest for data-hungry deep learning-based strategies. Furthermore, we present the up-to-date cell segmentation and tracking leaderboards, an in-depth analysis of the relationship between the performance of the state-of-the-art methods and the properties of the datasets and annotations, and two novel, insightful studies about the generalizability and the reusability of top-performing methods. These studies provide critical practical conclusions for both developers and users of traditional and machine learning-based cell segmentation and tracking algorithms.

LARNet-STC: Spatio-temporal orthogonal region selection network for laryngeal closure detection in endoscopy videos.

The vocal folds (VFs) are a pair of muscles in the larynx that play a critical role in breathing, swallowing, and speaking. VF function can be adversely affected by various medical conditions including head or neck injuries, stroke, tumor, and neurological disorders. In this paper, we propose a deep learning system for automated detection of laryngeal adductor reflex (LAR) events in laryngeal endoscopy videos to enable objective, quantitative analysis of VF function. The proposed deep learning system incorporates our novel orthogonal region selection network and temporal context. This network learns to directly map its input to a VF open/close state without first segmenting or tracking the VF region. This one-step approach drastically reduces manual annotation needs from labor-intensive segmentation masks or VF motion tracks to frame-level class labels. The proposed spatio-temporal network with an orthogonal region selection subnetwork allows integration of local image features, global image features, and VF state information in time for robust LAR event detection. The proposed network is evaluated against several network variations that incorporate temporal context and is shown to lead to better performance. The experimental results show promising performance for automated, objective, and quantitative analysis of LAR events from laryngeal endoscopy videos with over 90% and 99% F1 scores for LAR and non-LAR frames respectively.

Multi-focus Image Fusion for Confocal Microscopy Using U-Net Regression Map.

Characterizing the spatial relationship between blood vessel and lymphatic vascular structures, in the mice dura mater tissue, is useful for modeling fluid flows and changes in dynamics in various disease processes. We propose a new deep learning-based approach to fuse a set of multi-channel single-focus microscopy images within each volumetric z-stack into a single fused image that accurately captures as much of the vascular structures as possible. The red spectral channel captures small blood vessels and the green fluorescence channel images lymphatics structures in the intact dura mater attached to bone. The deep architecture Multi-Channel Fusion U-Net (MCFU-Net) combines multi-slice regression likelihood maps of thin linear structures using max pooling for each channel independently to estimate a slice-based focus selection map. We compare MCFU-Net with a widely used derivative-based multi-scale Hessian fusion method [8]. The multi-scale Hessian-based fusion produces dark-halos, non-homogeneous backgrounds and less detailed anatomical structures. Perception based no-reference image quality assessment metrics PIQUE, NIQE, and BRISQUE confirm the effectiveness of the proposed method.

Fluid and protein exchange in microvascular networks: Importance of modelling heterogeneity in geometrical and biophysical properties.

KEY POINTS: Microvascular network architecture defines coupling of fluid and protein exchange. Network arrangements markedly reduce capillary hydrostatic pressures and resting fluid movement at the same time as increasing the capacity for change The presence of vascular remodelling or angiogenesis puts constraints of network behaviour The sites of fluid and protein exchange can be segregated to different portions of the network Although there is a net filtration of fluid from a network of exchange vessels, there are specific areas where fluid moves into the circulation (reabsorption) and, when protein is moving into tissue, the amount is insufficient under basal conditions to result in changes in oncotic pressure. ABSTRACT: Integration of functional results obtained across scales, from chemical signalling to the whole organism, is a daunting task requiring the marriage of experimental data with mathematical modelling. In the present study, a novel coupled computational fluid dynamics model is developed incorporating fluid and protein transport using measurements in an in vivo frog (Rana pipiens) mesenteric microvascular network. The influences of network architecture and exchange are explored systematically under the common assumptions of structurally and functionally identical microvessels (Homogeneous Scenario) or microvessels classified by position in flow (Class Uniform Scenario), which are compared with realistic microvascular network components (Heterogeneous Scenario). The model incorporates ten quantities that vary within a microvessel; pressure boundary conditions are calibrated against experimental measurements. The Homogeneous Scenario standard model showed that assuming a single 'typical' capillary hides the influence of vessels arranged into a network architecture, where capillary hydrostatic pressures (pT ) are reduced, resulting in both a nonuniform distribution of blood flow and reduced volume flow rate (Jf,T ). In the Class Uniform Scenario pT was further attenuated to produce a ∼60% reduction in Jf,T . Finally, the Heterogeneous Scenario, incorporating measures of individual vessel surface area, demonstrates additional lowering of pT from inlet values favouring a >70% reduction of Jf,T in the face of a ∼120% increase in protein movement into the tissues relative to the Homogeneous Scenario. Beyond the impacts of network architecture, an unanticipated finding was the influence of a blind-end microvessel on model convergence, indicating a profound influence of the largely unexplored dynamics of vascular remodelling on tissue perfusion.

Defective Neuronal Positioning Correlates With Aberrant Motor Circuit Function in Zebrafish.

Precise positioning of neurons resulting from cell division and migration during development is critical for normal brain function. Disruption of neuronal migration can cause a myriad of neurological disorders. To investigate the functional consequences of defective neuronal positioning on circuit function, we studied a zebrafish frizzled3a (fzd3a) loss-of-function mutant off-limits (olt) where the facial branchiomotor (FBM) neurons fail to migrate out of their birthplace. A jaw movement assay, which measures the opening of the zebrafish jaw (gape), showed that the frequency of gape events, but not their amplitude, was decreased in olt mutants. Consistent with this, a larval feeding assay revealed decreased food intake in olt mutants, indicating that the FBM circuit in mutants generates defective functional outputs. We tested various mechanisms that could generate defective functional outputs in mutants. While fzd3a is ubiquitously expressed in neural and non-neural tissues, jaw cartilage and muscle developed normally in olt mutants, and muscle function also appeared to be unaffected. Although FBM neurons were mispositioned in olt mutants, axon pathfinding to jaw muscles was unaffected. Moreover, neuromuscular junctions established by FBM neurons on jaw muscles were similar between wildtype siblings and olt mutants. Interestingly, motor axons innervating the interhyoideus jaw muscle were frequently defasciculated in olt mutants. Furthermore, GCaMP imaging revealed that mutant FBM neurons were less active than their wildtype counterparts. These data show that aberrant positioning of FBM neurons in olt mutants is correlated with subtle defects in fasciculation and neuronal activity, potentially generating defective functional outputs.

DMNet: Dual-Stream Marker Guided Deep Network for Dense Cell Segmentation and Lineage Tracking.

Accurate segmentation and tracking of cells in microscopy image sequences is extremely beneficial in clinical diagnostic applications and biomedical research. A continuing challenge is the segmentation of dense touching cells and deforming cells with indistinct boundaries, in low signal-to-noise-ratio images. In this paper, we present a dual-stream marker-guided network (DMNet) for segmentation of touching cells in microscopy videos of many cell types. DMNet uses an explicit cell marker-detection stream, with a separate mask-prediction stream using a distance map penalty function, which enables supervised training to focus attention on touching and nearby cells. For multi-object cell tracking we use M2Track tracking-by-detection approach with multi-step data association. Our M2Track with mask overlap includes short term track-to-cell association followed by track-to-track association to re-link tracklets with missing segmentation masks over a short sequence of frames. Our combined detection, segmentation and tracking algorithm has proven its potential on the IEEE ISBI 2021 6th Cell Tracking Challenge (CTC-6) where we achieved multiple top three rankings for diverse cell types. Our team name is MU-Ba-US, and the implementation of DMNet is available at, http://celltrackingchallenge.net/participants/MU-Ba-US/.

Ensemble of Deep Learning Cascades for Segmentation of Blood Vessels in Confocal Microscopy Images.

Detection, segmentation, and quantification of microvascular structures are the main steps towards studying microvascular remodeling. Combined with appropriate staining, confocal microscopy imaging enables exploration of the full 3D anatomical characteristics of microvascular systems. Segmentation of confocal microscopy images is a challenging task due to complexity of anatomical structures, staining and imaging issues, and lack of annotated training data. In this paper, we propose a deep learning system for robust segmentation of cranial vasculature of mice in confocal microscopy images. The proposed system is an ensemble of two deep-learning cascades consisting of two coarse-to-fine subnetworks with skip connections in between. One cascade aims to improve sensitivity, while the other aims to improve precision of the segmentation results. Our experiments on mice cranial vasculature showed promising results achieving segmentation accuracy of 92.02% and dice score of 81.45% despite being trained on very limited confocal microscopy data.

Extending U-Net Network for Improved Nuclei Instance Segmentation Accuracy in Histopathology Images.

Analysis of morphometric features of nuclei plays an important role in understanding disease progression and predict efficacy of treatment. First step towards this goal requires segmentation of individual nuclei within the imaged tissue. Accurate nuclei instance segmentation is one of the most challenging tasks in computational pathology due to broad morphological variances of individual nuclei and dense clustering of nuclei with indistinct boundaries. It is extremely laborious and costly to annotate nuclei instances, requiring experienced pathologists to manually draw the contours, which often results in the lack of annotated data. Inevitably subjective annotation and mislabeling prevent supervised learning approaches to learn from accurate samples and consequently decrease the generalization capacity to robustly segment unseen organ nuclei, leading to over- or under-segmentations as a result. To address these issues, we use a variation of U-Net that uses squeeze and excitation blocks (USE-Net) for robust nuclei segmentation. The squeeze and excitation blocks allow the network to perform feature recalibration by emphasizing informative features and suppressing less useful ones. Furthermore, we extend the proposed network USE-Net not to generate only a segmentation mask, but also to output shape markers to allow better separation of nuclei from each other particularly within dense clusters. The proposed network was trained, tested, and evaluated on 2018 MICCAI Multi-Organ-Nuclei-Segmentation (MoNuSeg) challenge dataset. Promising results were obtained on unseen data despite that the data used for training USE-Net was significantly small. The source code of the USE-Net is available at https://github.com/CIVA-Lab/USE-Net.

Deep Learning-Based Cell Detection and Extraction in Thin Blood Smears for Malaria Diagnosis.

Malaria is a major health threat caused by Plasmodium parasites that infect the red blood cells. Two predominant types of Plasmodium parasites are Plasmodium vivax (P. vivax) and Plasmodium falciparum (P. falciparum). Diagnosis of malaria typically involves visual microscopy examination of blood smears for malaria parasites. This is a tedious, error-prone visual inspection task requiring microscopy expertise which is often lacking in resource-poor settings. To address these problems, attempts have been made in recent years to automate malaria diagnosis using machine learning approaches. Several challenges need to be met for a machine learning approach to be successful in malaria diagnosis. Microscopy images acquired at different sites often vary in color, contrast, and consistency caused by different smear preparation and staining methods. Moreover, touching and overlapping cells complicate the red blood cell detection process, which can lead to inaccurate blood cell counts and thus incorrect parasitemia calculations. In this work, we propose a red blood cell detection and extraction framework to enable processing and analysis of single cells for follow-up processes like counting infected cells or identifying parasite species in thin blood smears. This framework consists of two modules: a cell detection module and a cell extraction module. The cell detection module trains a modified Channel-wise Feature Pyramid Network for Medicine (CFPNet-M) deep learning network that takes the green channel of the image and the color-deconvolution processed image as inputs, and learns a truncated distance transform image of cell annotations. CFPNet-M is chosen due to its low resource requirements, while the distance transform allows achieving more accurate cell counts for dense cells. Once the cells are detected by the network, the cell extraction module is used to extract single cells from the original image and count the number of cells. Our preliminary results based on 193 patients (including 148 P. Falciparum infected patients, and 45 uninfected patients) show that our framework achieves cell count accuracy of 92.2%.

Clustering-Based Dual Deep Learning Architecture for Detecting Red Blood Cells in Malaria Diagnostic Smears.

Computer-assisted algorithms have become a mainstay of biomedical applications to improve accuracy and reproducibility of repetitive tasks like manual segmentation and annotation. We propose a novel pipeline for red blood cell detection and counting in thin blood smear microscopy images, named RBCNet, using a dual deep learning architecture. RBCNet consists of a U-Net first stage for cell-cluster or superpixel segmentation, followed by a second refinement stage Faster R-CNN for detecting small cell objects within the connected component clusters. RBCNet uses cell clustering instead of region proposals, which is robust to cell fragmentation, is highly scalable for detecting small objects or fine scale morphological structures in very large images, can be trained using non-overlapping tiles, and during inference is adaptive to the scale of cell-clusters with a low memory footprint. We tested our method on an archived collection of human malaria smears with nearly 200,000 labeled cells across 965 images from 193 patients, acquired in Bangladesh, with each patient contributing five images. Cell detection accuracy using RBCNet was higher than 97 %. The novel dual cascade RBCNet architecture provides more accurate cell detections because the foreground cell-cluster masks from U-Net adaptively guide the detection stage, resulting in a notably higher true positive and lower false alarm rates, compared to traditional and other deep learning methods. The RBCNet pipeline implements a crucial step towards automated malaria diagnosis.

Malaria Screener: a smartphone application for automated malaria screening.

BACKGROUND: Light microscopy is often used for malaria diagnosis in the field. However, it is time-consuming and quality of the results depends heavily on the skill of microscopists. Automating malaria light microscopy is a promising solution, but it still remains a challenge and an active area of research. Current tools are often expensive and involve sophisticated hardware components, which makes it hard to deploy them in resource-limited areas. RESULTS: We designed an Android mobile application called Malaria Screener, which makes smartphones an affordable yet effective solution for automated malaria light microscopy. The mobile app utilizes high-resolution cameras and computing power of modern smartphones to screen both thin and thick blood smear images for P. falciparum parasites. Malaria Screener combines image acquisition, smear image analysis, and result visualization in its slide screening process, and is equipped with a database to provide easy access to the acquired data. CONCLUSION: Malaria Screener makes the screening process faster, more consistent, and less dependent on human expertise. The app is modular, allowing other research groups to integrate their methods and models for image processing and machine learning, while acquiring and analyzing their data.

Robust mosaicking of maize fields from aerial imagery.

PREMISE: Aerial imagery from small unmanned aerial vehicle systems is a promising approach for high-throughput phenotyping and precision agriculture. A key requirement for both applications is to create a field-scale mosaic of the aerial imagery sequence so that the same features are in registration, a very challenging problem for crop imagery. METHODS: We have developed an improved mosaicking pipeline, Video Mosaicking and summariZation (VMZ), which uses a novel two-dimensional mosaicking algorithm that minimizes errors in estimating the transformations between successive frames during registration. The VMZ pipeline uses only the imagery, rather than relying on vehicle telemetry, ground control points, or global positioning system data, to estimate the frame-to-frame homographies. It exploits the spatiotemporal ordering of the image frames to reduce the computational complexity of finding corresponding features between frames using feature descriptors. We compared the performance of VMZ to a standard two-dimensional mosaicking algorithm (AutoStitch) by mosaicking imagery of two maize (Zea mays) research nurseries freely flown with a variety of trajectories. RESULTS: The VMZ pipeline produces superior mosaics faster. Using the speeded up robust features (SURF) descriptor, VMZ produces the highest-quality mosaics. DISCUSSION: Our results demonstrate the value of VMZ for the future automated extraction of plant phenotypes and dynamic scouting for crop management.

RTIP: A FULLY AUTOMATED ROOT TIP TRACKER FOR MEASURING PLANT GROWTH WITH INTERMITTENT PERTURBATIONS.

RTip is a tool to quantify plant root growth velocity using high resolution microscopy image sequences at sub-pixel accuracy. The fully automated RTip tracker is designed for high-throughput analysis of plant phenotyping experiments with episodic perturbations. RTip is able to auto-skip past these manual intervention perturbation activity, i.e. when the root tip is not under the microscope, image is distorted or blurred. RTip provides the most accurate root growth velocity results with the lowest variance (i.e. localization jitter) compared to six tracking algorithms including the top performing unsupervised Discriminative Correlation Filter Tracker and the Deeper and Wider Siamese Network. RTip is the only tracker that is able to automatically detect and recover from (occlusion-like) varying duration perturbation events.

Complex Non-sinus-associated Pachymeningeal Lymphatic Structures: Interrelationship With Blood Microvasculature.

The contribution of cranial dura mater vascular networks, as means for maintaining brain fluid movement and balance, and as the source of significant initiators and/or contributors to neurological disorders, has been overlooked. These networks consist of both blood and lymphatic vessels. The latter were discovered recently and described as sinus-associated structures thus changing the old paradigm that central nervous system lacks lymphatics. In this study, using markers specific to blood and lymphatic endothelia, we demonstrate the existence of the complex non-sinus-associated pachymeningeal lymphatic vasculature. We further show the interrelationship and possible connections between lymphatic vessels and the dural blood circulatory system. Our novel findings reveal the presence of lymphatic-like structures that exist on their own and/or in close proximity to microvessels. Of particular interest are sub-sets of vascular complexes with dual (lymphatic and blood) vessel identity representing a unique microenvironment within the cranial dura. The close association of the systemic blood circulation and meningeal lymphatics achieved in these complexes could facilitate fluid exchange between the two compartments and constitute an alternative route for CSF drainage.

Multiscale Structure Tensor for Improved Feature Extraction and Image Regularization.

Regularization methods are used widely in image selective smoothing and edge preserving restoration of noisy images. Traditional methods utilize image gradients within regularization function for controlling the smoothing and can produce artifacts when noise levels are higher. In this paper, we consider a robust image adaptive exponent driven regularization for filtering noisy images with salient feature preservation. Our spatially adaptive variable exponent function depends on a continuous switch based on the eigenvalues of structure tensor which identifies noisy edges, and corners with higher accuracy. Structure tensor eigenvalues encode various image features and we consider a spatially varying continuous map which provides multiscale edge maps of natural images. By embedding the structure tensor-based exponent in a well-defined regularization model, we obtain denoising filters which are capable of obtaining good feature preserving image restoration. The GPU-based implementation computes the edge map in real time at 45-60 frames/s depending on the GPU card. Multiscale structure tensor-based spatially adaptive variable exponent provides reliable edge maps and compared with standard edge detectors it is robust under various noisy conditions. Moreover, filtering based on the multiscale variable exponent map method outperforms L0 sparse gradient-based image smoothing and related filters.

Facial expression recognition for monitoring neurological disorders based on convolutional neural network.

Facial expressions are a significant part of non-verbal communication. Recognizing facial expressions of people with neurological disorders is essential because these people may have lost a significant amount of their verbal communication ability. Such an assessment requires time consuming examination involving medical personnel, which can be quite challenging and expensive. Automated facial expression recognition systems that are low-cost and noninvasive can help experts detect neurological disorders. In this study, an automated facial expression recognition system is developed using a novel deep learning approach. The architecture consists of four-stage networks. The first, second and third networks segment the facial components which are essential for facial expression recognition. Owing to the three networks, an iconize facial image is obtained. The fourth network classifies facial expressions using raw facial images and iconize facial images. This four-stage method combines holistic facial information with local part-based features to achieve more robust facial expression recognition. Preliminary experimental results achieved 94.44% accuracy for facial expression recognition on RaFD database. The proposed system produced 5% improvement than the facial expression recognition system by using raw images. This study presents a quantitative, objective and non-invasive facial expression recognition system to help in the monitoring and diagnosis of neurological disorders influencing facial expressions.

Sensitivity of Cross-Trained Deep CNNs for Retinal Vessel Extraction.

Automatic segmentation of vascular network is a critical step in quantitatively characterizing vessel remodeling in retinal images and other tissues. We proposed a deep learning architecture consists of 14 layers to extract blood vessels in fundoscopy images for the popular standard datasets DRIVE and STARE. Experimental results show that our CNN characterized by superior identifying for the foreground vessel regions. It produces results with sensitivity higher by 10% than other methods when trained by the same data set and more than 1% with cross training (trained on DRIVE, tested with STARE and vice versa). Further, our results have better accuracy $> 0 .95$% compared to state of the art algorithms.

Patch-Based Semantic Segmentation for Detecting Arterioles and Venules in Epifluorescence Imagery.

Segmentation and quantification of microvasculature structures are the main steps toward studying microvasculature remodeling. The proposed patch based semantic architecture enables accurate segmentation for the challenging epifluorescence microscopy images. Our pixel-based fast semantic network trained on random patches from different epifluorescence images to learn how to discriminate between vessels versus nonvessels pixels. The proposed semantic vessel network (SVNet) relies on understanding the morphological structure of the thin vessels in the patches rather than considering the whole image as input to speed up the training process and to maintain the clarity of thin structures. Experimental results on our ovariectomized - ovary removed (OVX) - mice dura mater epifluorescence microscopy images shows promising results in both arteriole and venule part. We compared our results with different segmentation methods such as local, global thresholding, matched based filter approaches and related state of the art deep learning networks. Our overall accuracy (> 98%) outperforms all the methods including our previous work (VNet). [1].

Malaria parasite detection and cell counting for human and mouse using thin blood smear microscopy.

Despite the remarkable progress that has been made to reduce global malaria mortality by 29% in the past 5 years, malaria is still a serious global health problem. Inadequate diagnostics is one of the major obstacles in fighting the disease. An automated system for malaria diagnosis can help to make malaria screening faster and more reliable. We present an automated system to detect and segment red blood cells (RBCs) and identify infected cells in Wright-Giemsa stained thin blood smears. Specifically, using image analysis and machine learning techniques, we process digital images of thin blood smears to determine the parasitemia in each smear. We use a cell extraction method to segment RBCs, in particular overlapping cells. We show that a combination of RGB color and texture features outperforms other features. We evaluate our method on microscopic blood smear images from human and mouse and show that it outperforms other techniques. For human cells, we measure an absolute error of 1.18% between the true and the automatic parasite counts. For mouse cells, our automatic counts correlate well with expert and flow cytometry counts. This makes our system the first one to work for both human and mouse.

Multiview Boosting With Information Propagation for Classification.

Multiview learning has shown promising potential in many applications. However, most techniques are focused on either view consistency, or view diversity. In this paper, we introduce a novel multiview boosting algorithm, called Boost.SH, that computes weak classifiers independently of each view but uses a shared weight distribution to propagate information among the multiple views to ensure consistency. To encourage diversity, we introduce randomized Boost.SH and show its convergence to the greedy Boost.SH solution in the sense of minimizing regret using the framework of adversarial multiarmed bandits. We also introduce a variant of Boost.SH that combines decisions from multiple experts for recommending views for classification. We propose an expert strategy for multiview learning based on inverse variance, which explores both consistency and diversity. Experiments on biometric recognition, document categorization, multilingual text, and yeast genomic multiview data sets demonstrate the advantage of Boost.SH (85%) compared with other boosting algorithms like AdaBoost (82%) using concatenated views and substantially better than a multiview kernel learning algorithm (74%).